Feat: add submit api stale alert and dashboard pannel - #68
Merged
Conversation
…panel Mirrors the Ogmios tip-stale pattern for the Submit API. The Submit API is a stateless proxy with no chain tip of its own, so staleness is detected via the backing cardano-node it forwards to: changes(cardano_node_metrics_blockNum_int) over 15m. When the node's block height stops advancing while the node is up, the Submit API is relaying transactions against a frozen tip. Adds grafana/alerts/demeter/submit-api.json (group "Submit API", folder "Demeter", rule "Submit API backing node is stale", for 10m, receiver #infra-alerts) and a "Node Block Advancement (changes / 15m)" timeseries panel (id 20) to the SubmitAPI dashboard. Signed-off-by: Tripura Repalle <tripura@blinklabs.io>
The grafana_alerts module only created a folder when the configured folder_uid was empty, but config.yaml always supplies a concrete UID (e.g. de9p56uby56v4a for Demeter). As a result the folder was never created, and terraform apply failed for every rule group with '[PUT .../rule-groups/...][400] invalid alert rule: folder does not exist'. Now the module always manages its folder with the config.yaml title/uid (mirroring the grafana_dashboard module), guarded by a count so a missing/empty alert directory (e.g. grafana/alerts/blinklabs) stays a no-op and never creates a stray folder. Rule groups reference the managed folder's uid. Removes the now-unused folders variable. Signed-off-by: Tripura Repalle <tripura@blinklabs.io>
wolf31o2
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #67
Summary
Adds staleness monitoring for the Submit API (alert + dashboard panel) and fixes a bug that prevented any Demeter alert rules from being provisioned.
Submit API staleness
The Submit API is a stateless HTTP proxy that forwards transaction submissions to a
cardano-node; it exposes no chain tip of its own. If the node it forwards to has a frozen chain tip, submissions are relayed against a stale tip with no visibility today. This PR detects that condition via the backing node's block-height progress.grafana/alerts/demeter/submit-api.jsonSubmit API, folderDemeter, rule "Submit API backing node is stale",for: 10m, receiver#infra-alerts.changes(cardano_node_metrics_blockNum_int{namespace="ext-nodes-m1", pod=~"node-.*"}[15m]) < 1— fires when the backing node is up but its block height has not advanced for at least 15 minutes.grafana/dashboards/shared/submit-api.json), red@0 / green@1, filtered by$network.This is complementary to the existing
node.jsonalert: that one detects a node whose series disappears, whereas this detects a node that is up but frozen.Fix: alert folder was never created
Instantiating the
grafana_alertsmodule surfaced a pre-existing bug: the module only created its folder when the configuredfolder_uidwas empty, butconfig.yamlalways supplies a concrete UID (e.g.de9p56uby56v4afor Demeter). As a result the folder was never created, and every rule group failed on apply with:The module now always manages its folder with the
config.yamltitle/uid (mirroring thegrafana_dashboardmodule), guarded by acountso a missing/empty alert directory (e.g.grafana/alerts/blinklabs) stays a no-op and never creates a stray folder. Rule groups reference the managed folder's uid, and the now-unusedfoldersvariable is removed.Validation
terraform fmtclean andterraform validatepasses. Aplanon this branch is clean:cluster,fabric,kupo,kupo-dolos,node,submit-api)